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CHAPTER  1 


INTRODUCTION 

Background 

In  an  era  of  dwindling  defense  budgets  and  scarce  resources,  maintaining  and 
preserving  combat  power  for  the  nation's  defense  is  an  important  task  for  the  United 
States  Air  Force  (USAF).  Since  fiscal  year  (FY)  2000,  the  USAF  has  lost  102  fighter 
aircraft  at  a  cost  of  over  $2.4  billion  in  various  mishaps  and  aircraft  accidents. 1  More 
importantly,  the  USAF  also  lost  32  fighter  pilots.  While  the  cost  of  training  replacements 
for  the  32  lost  airmen  exceed  $83  million,2  the  pain  and  suffering  of  the  families  and 
units  of  those  ainnen  cannot  be  calculated.  The  USAF  aircraft  maintenance  community 
must  do  all  it  can  to  prevent  aircraft  mishaps  and  retain  both  the  human  and  physical 
capital  invested  in  our  nation's  fighter  force. 

The  primary  job  of  an  USAF  aircraft  maintenance  organization  is  to  provide  safe, 
serviceable,  and  properly  configured  aircraft  to  the  pilots  who  fly  them/  This  task  is  the 
backbone  of  airpower  production  and  cannot  be  executed  haphazardly.  Maintenance  tasks 
perfonned  on  ground  vehicles,  computer  systems,  radar  systems,  and  many  other  systems 
are  equally  important  to  mission  accomplishment.  Failure  in  tasks  outside  of  aircraft 

’U.S.  Air  Force  Safety  Center,  “F-16  Aircraft  Statistics,”  http://www.afsc.af.mil/ 
organizations/aviation/aircraftstatistics/index.asp  (accessed  24  February  2012). 

‘Eric  Tegler,  “Air  Force  Flight  Simulators  May  Help  Cut  Training  Costs,” 
Defense  Media  Network,  1 1  November  2011,  http://www.defensemedianetwork.com/ 
stories/virtual-bargain/  (accessed  3  March  2012). 

3 

Department  of  the  Air  Force,  Air  Force  Instruction  (AFI)  21-101,  Aircraft  and 
Equipment  Maintenance  Management,  para  1.3.  (Washington,  DC:  Government  Printing 
Office,  2010),  www. e.publishing. af.mil  (accessed  15  May  2012). 
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maintenance  can  lead  to  mission  degradation,  in  various  areas  of  communication, 
information  flow,  and  logistical  capabilities.  Failure  to  adhere  to  technical  orders 
published  for  aircraft  maintenance  can  lead  to  catastrophic  and  sometimes  fatal  results. 

In  an  incident  in  March  2001,  an  F-16C  flying  over  a  training  range  in  New 
Mexico  lost  thrust,  forcing  the  pilot  to  eject  and  abandon  a  $30-$35  million  jet.  An 
investigation  detennined  that  faulty  maintenance  actions  were  responsible  for  the  engine 
failure.4  In  a  similar  incident  in  April  2005,  an  F-16D  was  flying  a  training  mission  near 
Charleston,  South  Carolina.  After  the  engine  lost  thrust,  the  pilots  on  board  steered  the 
aircraft  away  from  the  city  and  ejected  prior  to  the  aircraft  crashing  into  the  Charleston 
marshland.  It  was  subsequently  detennined  that  key  blade  seals  were  not  installed  on  the 
engine’s  turbine  section.  5  These  blade  seals  are  a  vital  part  of  the  engine's  turbine  section 
and  necessary  to  maintain  appropriate  air  pressure  and  airflow  through  the  engine.  They 
prevent  air  from  leaking  through  gaps,  ensuring  the  efficiency  and  thrust  of  the  motor. 
This  is  especially  important  in  a  single-engine  aircraft  like  the  F-16. 

From  FY  2001  through  2006,  a  total  of  ten  Class  A  mishaps  were  directly 
attributed  to  aircraft  maintenance  failures.  Since  FY  2007,  the  USAF  has  grouped 
maintenance  and  logistics  into  a  single  category  for  Class  A  mishap  tracking.  Since  that 
time  the  USAF  has  determined  34  Class  A  mishaps  were  a  result  of  maintenance  and 
logistics.6  Combining  these  two  areas  highlights  the  need  to  focus  efforts,  not  only  across 

4Miguel  Navrot,  “Workers  blamed  in  F-16  Crash,”  Albuquerque  Journal,  3 1 
August  2001. 

5Ron  Menchaca,  “F-16D  laced  seals,  crash  report  finds,”  The  Post  and  Courier, 

24  August  2005. 

6U.S.  Air  Force  Safety  Center,  “F-16  Aircraft  Statistics.” 
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the  maintenance  enterprise  from  the  lowest  organizational  levels  back  to  depot 
maintenance  levels,  but  also  through  supply  and  support  chains,  to  prevent  mishaps. 

The  USAF  devotes  significant  resources  to  investigating  and  preventing  mishaps. 
It  offers  eight  different  courses  devoted  to  aviation  safety,  investigation,  and  prevention 
that  target  officers  and  senior  non-commissioned  officers.  The  USAF  also  trains 
international  students  on  mishap  prevention  and  investigation.7  Each  flying  wing  has  a 
Chief  of  Safety  position,  as  well  as  dedicated  Flying  Safety  Officers  devoted  to  aviation 
mishap  prevention  and  responsible  for  the  initial  stages  of  an  investigation  should  one  be 
required.  The  Chief  of  Safety  position  is  important  enough  to  have  a  prerequisite 
assigned,  requiring  the  individual  assigned  to  the  position  to  currently  be  on  or  have  been 
on  a  previous  squadron  commander  list.  If  the  individuals  Wing  Commander  intends  to 
support  his  or  her  nomination  to  the  next  squadron  commander  list  he  or  she  would  also 
be  eligible  for  this  important  position.  This  requirement  underscores  the  value  the  USAF 
places  on  the  position  and  the  caliber  of  individual’s  desired  to  fill  front-line  safety 
positions.8 

While  a  culture  of  maintenance  discipline  and  proper  maintenance  actions  is 
difficult,  if  not  impossible  to  fully  quantify,  this  paper  seeks  to  identify  measures  that 
may  prove  useful  in  determining  if  a  given  unit  is  headed  toward  a  catastrophic  mishap. 
Many  arguments  can  be  supported  with  respect  to  maintenance  culture  being  the  most 
important  factor  in  preventing  mishaps.  A  “by  the  book”  mentality  is  what  every 
7Ibid. 

department  of  the  Air  Force,  Air  Force  Instruction  (AFI)  91-202,  The  US  Air 
Force  Mishap  Prevention,  para  2. 1.1.1.  (Washington,  DC:  Government  Printing  Office, 
2011),  www.e-publishing.af.mil  (accessed  15  May  2012). 
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maintenance  leader  strives  for;  however,  quality  assurance  functions  and  mishap  reports 
often  reveal  a  failure  to  adhere  to  technical  maintenance  requirements  or  orders.  The 
safety  investigation  process  searches  for  the  cause  or  causes  of  a  mishap,  often  focusing 
on  material  failure,  pilot  error,  or  failure  of  maintenance  personnel  to  perform  their  jobs 
correctly.  This  paper  seeks  to  find  correlation  between  maintenance  metrics  and  the 
number  of  unit  mishaps. 

Aircraft  maintenance  perfonnance  metrics  have  been  tracked  since  the  inception 
of  the  USAF.9  The  requirement  to  maintain  our  nation’s  fleet  of  combat  aircraft  is 
arguably  the  lynchpin  for  successful  operations.  Many  operations  depend  on  the  ability  of 
the  USAF  to  provide  strategic  airlift  across  the  globe.  Reliable  and  safe  aircraft 
maintenance  is  the  cornerstone  of  the  USAF  maintenance  community.  In  2001  the  USAF 
produced  the  Metrics  Handbook  for  Maintenance  Leaders. 10  This  relatively  short 
handbook  captured  and  explained  many  of  the  pertinent  maintenance  metrics,  how  to 
interpret  them,  and  the  importance  of  trends  in  a  given  area  of  measurement  of  aircraft 
maintenance.  Dividing  the  metrics  into  leading  and  lagging  indicators,  this  pamphlet 
provides  the  basis  for  rating  an  aircraft  maintenance  unit’s  perfonnance  over  time,  in 
specific  and  quantifiable  measures.  These  metrics,  the  definitions  of  the  processes  they 
measure,  and  their  applicability  to  maintenance  management  will  be  explained  in  detail  in 
chapter  4.  One  unit’s  apparently  positive  performance  metrics  measured  against  a  higher 

9Barbara  L.  Harris,  “Challenge  to  United  States  Tactical  Air  Force  Aircraft 
Maintenance  Personnel:  Past,  Present  and  Future”  (Thesis,  Air  Force  Institute  of 
Technology,  1991),  44. 

10Air  Force  Logistics  Management  Agency,  Metrics  Handbook  for  Maintenance 
Leaders,  http://www.aflma.hq.af.mil/shared/media/document/AFD-100107-055.pdf 
(accessed  15  May  2012),  2. 
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headquarters  mandated  scale  is  one  indication  of  performance;  however,  when  measured 
against  the  unit’s  number  of  mishaps  experienced,  it  may  present  a  different  picture. 
Additionally,  chapter  four  will  describe  USAF  mishap  reporting  categories  and  classes  to 
provide  context  to  the  reader.  Chapter  5  will  explain  findings  and  their  significance. 

Problem  Statement 

The  USAF  is  entering  a  period  of  fiscal  constraints  as  the  national  economy 
forces  the  entire  country  to  reexamine  priorities  and  expenditures.  As  major  weapons 
systems  acquisition  programs  are  curtailed,  reduced,  or  eliminated,  the  need  to  preserve 
and  maintain  current  combat  aircraft  becomes  more  important  than  ever.  The  challenge  to 
the  USAF  is  figuring  out  how  to  utilize  maintenance  metrics  to  prevent  aircraft  mishaps. 
The  purpose  of  this  thesis  will  be  to  assess  if  any  correlation  exists  between  its 
maintenance  performance  metrics,  and  the  unit’s  mishap  record. 

Research  Question 

Specifically,  this  thesis  will  attempt  to  answer  this  question:  Does  a  discernible, 
statistically  relevant  relationship  exist  between  any  one  of  nine  specific  active-duty  F-16 
maintenance  performance  indicators  and  active-duty  F-16  mishaps? 

Scope  and  Limitations 

The  scope  of  the  paper  was  purposefully  focused  on  F-16  aircraft  based  upon  the 
author’s  personal  experience  with  the  airframe,  the  availability  of  data  for  the  aircraft, 
and  the  sample  size  of  the  number  of  units  flying  and  maintaining  the  aircraft.  The  F-16 
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has  also  been  in  service  since  1979,  providing  a  reasonable  data  sample. 1 1  The  scope  of 
research  for  this  paper  will  be  restricted  to  current  active  duty,  F-16  units  for  the  period 
of  FY  2000  through  FY  2008.  The  reason  for  ending  the  study  at  FY  2008  was  due  to  a 
Class  A  mishap  criteria  change  across  the  Department  of  Defense,  raising  the  threshold 
for  Class  A  monetary  losses  to  two  million  dollars.  In  order  to  keep  the  comparisons 
equal,  the  sample  of  active  duty  unit’s  maintenance  metrics  was  restricted  to  FYs  2000 
through  2008.  Additionally,  units  deactivated,  or  given  a  new  mission  or  aircraft  mission 
design  series  will  not  be  reviewed  due  to  a  shortage  of  data  from  the  time  period  studied. 
Air  National  Guard  and  Air  Force  Reserve  units  on  Title  10  orders  deployed  or 
participating  in  training  exercises,  while  technically  considered  active  duty,  will  also  be 
excluded  to  maintain  a  comparable  data  sets.  Deployed  aircraft  mishaps  are  also  removed 
from  the  data  set,  regardless  of  the  unit  maintaining  those  aircraft,  because  the  reporting 
tool  consistently  fails  to  identify  the  specific  home  unit  in  a  deployed  setting. 

The  reason  for  excluding  Air  National  Guard  and  Reserve  data  is  to  ensure 
comparable  data  set.  To  further  illustrate  this  reasoning,  active-duty  units  are  traditionally 
more  focused  on  the  “holy  grail”  of  maintenance  metrics,  the  Mission  Capable  (MC)  rate, 
for  it  describes  from  one  point  of  view  the  degree  to  which  the  unit  is  successful  in 
maintaining  assigned  aircraft.  The  MC  rate  traditionally  has  been  the  yardstick  for 


1  'inside  AF.mil,  F-16  Fighting  Falcon  fact  sheet,  http://www.af.mil/information/ 
factsheets/factsheet.asp?id=103  (accessed  17  May  2012). 

12Department  of  the  Air  Force,  Memorandum  for  All  MAJCOM/SE  and  All 
NAF/SE,  Subject:  AFI  91-204  and  AFMAN  91-223  Immediate  Action  Guidance,  14 
October  2009. 
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measuring  a  unit’s  performance.  While  it  does  show  the  number  of  flyable  aircraft  and 
condition  of  those  aircraft  may  be  mission  capable,  they  may  not  be  in  the  best  possible 
condition  for  long-term  fleet  health.  Guard  units  not  as  focused  on  the  MC  rate  (or  many 
maintenance  metrics  to  the  extent  the  active  duty  is)  may  devote  extra  time  to  fix  pilot 
reported  or  ground  discovered  discrepancies.  Another  factor  making  the  active  duty 
versus  guard  comparison  problematic  is  the  different  standards  to  which  each  are  held. 
For  example,  in  FY  2005  the  USAF  F-16  MC  rate  standard  was  82  percent  for  active 
duty  units  and  72  percent  for  Air  National  Guard  and  Air  Force  Reserve  units. 14  The 
previous  FY  (2004)  the  active  duty  units  maintained  their  F-16  fleets  at  82.4  percent  MC 
rate  while  their  guard  and  reserve  counterparts’  fleets  were  maintained  at  68.6  percent 
MC.  This  almost  14  percent  difference  is  the  result  of  many  factors,  but  the  difference  in 
the  two  rates  makes  comparisons  problematic.  Additionally,  the  Total  Non-Mission 
Capable  Maintenance  (TNMCM)  rates  for  the  active  duty  versus  guard  and  reserve  units 
for  FY  2004  were  12.2  to  27.8  respectively.  The  subsequent  TNMCM  standard  for  FY 
2005  was  set  for  active  duty  units  at  1 1  percent  TNMCM,  while  the  guard  and  reserve 
units  were  allowed  a  28  percent  TNMCM  standard.15  This  17  percent  difference  in 
standards  allowed  guard  and  reserve  units  significantly  more  time  to  allow  aircraft  to 
remain  in  a  non-mission  capable  status  based  solely  on  maintenance  requirements.  The 

13* 

Air  Force  Logistics  Management  Agency,  Metrics  Handbook  for  Maintenance 
Leaders,  http://www.aflma.hq.af.mil/shared/media/document/AFD-100107-055.pdf 
(accessed  15  May  2012),  40. 

14Christopher  Burke,  Operational  Based  AF  Maintenance  Standards,  www.acq. 
osd.mil/07_USAF%20MX%20Standards%20Brief3.ppt  (accessed  17  May  2012). 

15Ibid. 
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effect  of  these  different  standards  translates  into  more  pressure  on  active  duty  units  to 
meet  a  higher  standard  of  readiness  and  therefore  would  not  represent  a  fair  comparison 
of  the  units  maintenance  metrics  and  number  of  aircraft  mishaps  experienced. 

The  use  of  aggregate  USAF  data  for  FYs  2001  through  2010  is  shown  only  as  a 
comparison  of  data  sets  and  as  an  overall  tool  for  understanding  the  entire  F-16 
community.  This  much  larger  data  set  includes  all  F-16  units  regardless  of  status  (active 
duty,  guard  or  reserve),  size,  or  mission. 

Expected  research  value 

The  ultimate  value  of  this  study  is  how  it  may  provide  maintenance  managers  at 
all  levels  and  possibly  the  USAF  in  general  with  trend  data  and  increased  awareness  of 
the  importance  of  maintenance  metrics  and  the  prevention  of  F-16  mishaps.  This  study  is 
not  intended  to  advocate  for  or  argue  against  the  value  of  maintenance  metrics.  The 
purpose  of  this  paper  is  to  study  the  metric’s  intrinsic  value  in  predicting  aircraft  mishaps 
through  an  examination  of  nine  specific  maintenance  metrics  and  whether  or  not  they 
correlate  to  the  number  of  mishaps  experienced  by  a  unit.  If  any  correlation  is  found 
between  maintenance  metrics  and  mishap  rates  in  the  study,  it  could  be  expanded  to  other 
airframes  in  an  effort  to  predict  future  aircraft  mishaps  across  the  entire  fleet  of  USAF 
aircraft. 
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CHAPTER  2 


LITERATURE  REVIEW 

Aviation  Mishap  Prevention 

With  respect  to  aviation  mishap  prevention,  extensive  literature  exists,  most  of 
which  is  focused  on  prevention  of  the  next  mishap  through  risk  mitigation,  increasing 
awareness,  and  using  lessons  learned  after  detennining  factors  that  caused  current 
mishaps.  In  the  United  States  the  Federal  Aviation  Administration,  the  Department  of  the 
Interior,  and  all  services  in  the  Department  of  Defense,  administer  and  resource  robust 
mishap  prevention  programs.  A  multitude  of  government  agencies  across  the  globe 
maintain  programs  aimed  at  preventing  aviation  mishaps.  The  International  Civil 
Aviation  Organization’s  website  lists  191  member  states  in  the  organization16  and  places 

17 

aviation  safety  at  the  core  of  its  objectives.  At  any  of  the  organization  websites  listed, 
safety  and  mishap  prevention  bulletins,  are  provided. 

Post  Mishap  Investigation  in  the  USAF 

The  standard  USAF  mishap  investigation  procedure,  as  outlined  in  Air  Force 
Instruction  91-204,  focuses  on  finding  the  failed  part,  procedure,  or  cultural  factor  so  that 
it  can  be  rectified  through  engineering,  maintenance  or  operations  technical  orders,  or 
through  changes  in  the  organization’s  culture.  As  stated  in  Air  Force  Pamphlet  91-211: 
“The  purpose  of  every  safety  investigation  is  to  detennine  all  factors  (human,  materiel, 

'^'International  Civil  Aviation  Organization,  ICAO  Member  States,  http:// 
www.icao.int/MemberStates/Member%20States.English.pdf  (accessed  17  May  2012). 

17 

International  Civil  Aviation  Organization,  Strategic  Objectives,  http:// 
www.icao.int/safety/Pages/default.aspx  (accessed  17  May  2012). 
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and  environmental)  that  directly  or  indirectly  contributed  to  the  mishap.  This  infonnation 

is  used  by  aircrews,  equipment  operators,  supervisors,  commanders,  staffs,  and  designers 

1 8 

to  eliminate  cause  factors  and  thus  help  prevent  recurrence  of  similar  mishaps.” 

Extensive  information  exists  regarding  F-16  maintenance  performance  indicators 
and  almost  all  other  USAF  aircraft  on  the  Fogistics  Installations  and  Mission  Support  - 
Enterprise  View  (FIMS-EV)  database.  This  database  serves  as  the  primary  source  for  all 
maintenance  metrics  for  this  thesis.  Through  its  Weapons  System  View,  it  provides 
USAF  leaders  a  one-stop-shop  for  supporting  details  and  a  single  source  for  viewing 
analytical  metrics  for  current  and  historical  weapon  systems  information. 19  Normally 
used  as  a  tool  for  identifying  focus  areas  for  management  efforts  in  sustaining  aircraft 
fleet  logistic  support  areas,  FIMS-EV  can  also  be  used  as  a  historical  database  to 
compare  past  perfonnance  indicators  and  units. 

F-16  mishap  data  is  available  from  the  USAF  Safety  Center  through  the  Air  Force 
Safety  Automated  System  or  AFSAS,  as  is  all  aircraft  mishap  data  for  the  USAF.  This 
data  is  used  throughout  the  USAF,  but  usually  is  most  effective  immediately  after  an 
incident  as  the  convening  authority  considers  any  recommendations  from  the  Safety 
Investigation  Board.  These  recommendations  often  suggest  changes  to  maintenance 
technical  orders  or  pilot  checklists  to  prevent  similar  situations  or  events  from  occurring 
again. 

1  department  of  the  Air  Force,  Air  Force  Pamphlet  91-211,  USAF  Guide  To 
Aviation  Safety  Investigation,  para  1.2.  (Washington,  DC:  Government  Printing  Office, 
2011),  www.e-publishing.af.mil  (accessed  15  May  2012). 

19Headquarters  United  States  Air  Force,  A4IS,  L1MS-EV 101  Brief 
http://www.acq.osd.mil/log/mpp/cbm-l-/Briefmgs/FIMS-EV_OSD_CBMPlus_AG_ 
Brief.pdf  (accessed  7  April  2012). 
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Predicting  Mishaps 

A  search  was  conducted  in  an  attempt  to  find  indicators  of  any  kind  past  or 
present,  used  in  aviation  mishap  prevention.  While  extensive  programmed  maintenance 
practices  exist  to  prevent  system  or  equipment  failure,  use  of  such  programs  is  aimed  at 
finding  failures  in  material  parts  and  systems,  usually  identified  by  past  failures  or 
mishaps.  Specific  literature  regarding  the  use  of  metrics  in  predictive  mishap  prevention 
exists  in  the  form  of  a  paper  I.  A.  Herrara  and  J.  Hovden  presented  at  the  3rd  Resilience 
Engineering  Symposium,  in  October  of  2008  in  Antibes- Juan  Les  Pins,  France.  This 
seven  page  paper,  titled  “Leading  Indicators  Applied  to  Maintenance  in  the  Framework 
of  Resilience  Engineering:  A  Conceptual  Approach”  specifically  focused  on  aviation 
maintenance  indicators  and  their  usefulness  in  mishap  prevention."  This  paper  identified 
desirable  characteristics  for  potential  indicators,  such  as  objective  measurement,  easy 
comprehension,  positive  or  negative  improvement  or  deterioration,  and  readily  available, 
while  failing  to  actually  identify  specific  indicators.  The  USAF  identifies  19  specific 
metrics  as  leading  indicators  and  13  metrics  as  lagging  indicators  for  the  maintenance 
manager.22 


20 

"  I.  A.  Herrera  and  J.  Hovden,  “Leading  indicators  applied  to  maintenance  in  the 
framework  of  resilience  engineering:  A  conceptual  approach”  (Paper  presented  at  The 
3rd  Resilience  Engineering  Symposium,  28-30  October  2008),  http://www.sintef.no/ 
project/Building%20Safety/Publications/2008-Resilience%20Engineering%20 
Symposium-Leading%20indicators-herrera-hovden.pdf  (accessed  15  May  2012). 

21Ibid. 

22 

"  Air  Force  Logistics  Management  Agency,  Metrics  Handbook  for  Maintenance 
Leaders,  http://www.aflma.hq.af.mil/shared/media/document/AFD-100107-055.pdf 
(accessed  15  May  2012),  15-16. 
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Herrera  and  Hovden  note,  “Traditionally,  improvements  in  aviation  have  been 
based  on  incident  reporting  analyses  and  learning  from  failures.  This  traditional  approach 
measures  safety  performance  based  on  lagging  indicators.”23  Their  search  for  a  predictive 
metric  is  a  rare  one.  It  is  also  the  purpose  of  this  thesis. 


23 

'  Herrera  and  Hovden,  “Leading  indicators  applied  to  maintenance  in  the 
framework  of  resilience  engineering:  A  conceptual  approach”  (Paper  presented  at  The 
3rd  Resilience  Engineering  Symposium,  28-30  October  2008),  http://www.sintef.no/ 
project/Building%20Safety/Publications/2008-Resilience%20Engineering%20 
Symposium-Leading%20indicators-herrera-hovden.pdf  (accessed  15  May  2012). 
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CHAPTER  3 


RESEARCH  METHODOLOGY 

Overview 

This  thesis  uses  a  quantitative  comparison  of  maintenance  performance  indicators 
for  all  active  duty  F-16  units  for  the  FYs  2000-08.  This  infonnation  will  be  compared  to 
the  mishap  rates  for  the  same  units  for  the  same  years  in  an  attempt  to  detennine  if  the 
data  shows  any  relevant  correlation  with  the  unit’s  Class  A  mishaps.  Following  a 
summarization  of  the  entire  USAF  F-16  fleet's  maintenance  metrics  versus  Class  A 
mishaps  from  FYs  2001  through  2010,  a  comparison  of  all  active  duty  F-16  unit 
performance  metrics  with  their  respective  mishaps  by  year  will  follow,  covering  FYs 
2000-08.  Unit  data  was  also  compiled  and  analyzed  for  applicability  and  subjected  to 
correlation  tests  to  determine  if  a  unit’s  perfonnance  metrics  versus  mishap  rate  was 
statistically  relevant  using  Pearson's  correlation  coefficient.  While  a  significant  number 
of  variables  exist  among  the  units  studied,  all  were  F-16  and  active-duty  from  FY  2000- 
OS. 


Data  Sources 

The  data  considered  for  this  thesis  is  derived  primarily  from  two  sources:  The  first 
is  the  USAF’s  Air  Force  Safety  Automated  System.  The  second  source  is  maintenance 
perfonnance  indicators  pulled  from  the  Logistics  Installations  and  Mission  Support  - 
Enterprise  View  (LIMS-EV)  system  of  databases.  These  systems  compile  relevant 
information  for  their  respective  areas  from  across  the  USAF.  A  discussion  of  the  meaning 
of  maintenance  performance  indicators  will  open  chapter  4  as  an  explanation  of  the  data 
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analysis  process  followed  throughout  the  study,  citing  data  pulled  from  LIMS-EV  and  the 
USAF  Maintenance  Metrics  pamphlet  and  appropriate  Air  Force  Instructions.  Any  holes 
or  incongruent  data  will  be  explained.  The  use  or  exclusion  of  a  specific  set  of  data  will 
also  be  explained  in  chapter  4. 

Air  Force  Safety  Automated  System 

Drawn  from  the  restricted  AFSAS  database,  reporting  tools  specifying  FYs, 
bases,  units,  and  mission  design  series  aircraft  were  used,  searching  only  for  Class  A 
mishaps.  These  comprehensive  reports  enabled  the  author  to  report  on  37  specific 
mishaps  from  over  69  Class  A  F-16  mishaps  from  the  FYs  studied.  The  limitations  and 
scope  of  this  paper  as  described  in  chapter  1  necessitated  the  reduction  in  data  and 
allowed  the  scope  of  study  to  be  maintained. 

Logistics,  Installations  and  Mission  Support-Enterprise  View 
LIMS-EV  data  was  provided  in  part  by  USAF  maintenance  analysts  and 
examined  for  specific  F-16  metrics,  focusing  on  the  most  commonly  used  metrics  in  the 
F-16  maintenance  community.  These  metrics  include:  Mission  Capable  Rate,  Total  Non- 
Mission  Capable  Maintenance  Rate,  Total  Non-Mission  Capable  Supply  Rate,  Break 
Rate,  8-hour  Fix  Rate,  Abort  Rate,  Repeat  and  Recurrence  Rate,  the  Flying  Scheduling 
Effectiveness  Rate,  and  the  Cannibalization  Rate.  The  data  from  LIMS-EV  is  98  percent 
accurate  and  complete  as  of  May  201 1. 24 


24Headquarters  United  States  Air  Force,  A4IS,  LIMS-EV  101  Brief, 
http://www.acq.osd.mil/log/mpp/cbm-l-/Briefmgs/LIMS-EV_OSD_CBMPlus_AG 
Brief.pdf  (accessed  7  April  2012). 
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sets  of  data  was  a  simple  scattergram,  commonly  used  in  statistics.  Dictionary.com  first 
defines  correlation  as:  mutual  relation  of  two  or  more  things,  parts,  etc.  The  second 
definition  is:  the  act  of  correlating  or  state  of  being  correlated.  In  its  third  definition,  it 
specifically  defines  correlation  with  respect  to  statistics  as  the  degree  to  which  two  or 
more  attributes  or  measurements  on  the  same  group  of  elements  show  a  tendency  to  vary 
together.  It  is  important  to  note  that  correlation  does  not  mean  causation.  The  first 
comparison  is  between  the  aggregate  maintenance  data  (entire  F-16  fleet)  and  the 
aggregate  mishap  data.  Each  metric  will  then  be  broken  down  to  compare  each  base’s 
maintenance  performance  in  the  nine  areas  against  its  mishap  perfonnance  rates  by  year 
and  for  the  entire  period.  Data  will  be  presented  in  graphs  and  tables  followed  by 
explanations  by  unit  in  line  with  the  aggregate  data. 


Correlation  determination 

Scattergrams  built  from  the  LIMS-EV  data  and  corresponding  mishaps  show 
trend  lines  and  corresponding  values  using  the  Pearson  correlation  coefficient.  The 
Pearson  coefficient  is  a  widely  accepted  basic  correlation  tool  used  in  statistics  and  the 
most  basic  type.26  While  the  computations  involved  in  the  manufacture  of  the  trend  line 
and  the  accompanying  coefficient  or  R2  number  were  generated  by  the  spreadsheet 
program,  the  results  indicate  how  likely  the  metric  is  related  to  the  number  of  mishaps 

“  Dictionary.com,  http://dictionary.reference.com/  (accessed  17  May  2012). 

26 

'  Creative  Research  Systems,  “Correlation,”  http://www.surveysystem.com/ 
correlation.htm  (accessed  17  May  2012). 


15 


experienced.  The  closer  the  coefficient  approaches  1  or  perfect  correlation,  the  more 
likely  the  two  sets  of  data  correlate.  The  data  for  the  scattergrams  was  arranged  in 
ascending  order,  from  lowest  value  for  the  metric  to  the  highest  value  for  the  years 
measured  and  the  number  of  mishaps  experienced  for  that  unit  at  that  metric  level  for  that 
respective  FY.  This  ensured  the  data  reflected  the  potential  correlation  the  metric  had  on 
the  mishap— or  lack  thereof.  The  resultant  coefficients  were  compared  across  the  entire 
USAF  F-16  fleet  for  FYs  2001  through  2010  against  all  Class  A  mishaps.  The 
comparison  across  each  of  the  nine  active  duty  bases  for  FYs  2000  through  2008,  was 
accomplished  by  arranging  the  correlation  coefficients  generated  by  each  unit,  for  each 
metric  for  the  nine  years  studied,  and  taking  the  average  of  the  coefficients  in  each 
category. 

The  statistical  relevance  of  the  correlation  coefficients  was  determined  by 
comparing  each  of  the  values  derived  to  a  critical  value  chart  for  Pearson’s  coefficient 
correlation.  For  the  sample  size  of  nine  sets  of  data  for  each  metric  measured  over  the 
nine  year  study  period,  the  degree  of  freedom  used  was  seven,  the  normal  N-2  figure 
where  N=the  sample  size  of  the  study,  in  this  case  the  number  of  years  being  used  as  the 

27 

sample  for  the  sets  of  data  used. 

Interpretation  of  Pearson  correlation  coefficients  varies,  but  generally  the 

28 

following  table  demonstrates  the  potential  correlation  of  factors  for  two  sets  of  data:- 

97 

"  Del  Siegle  Ph.D.,  Neag  School  of  Education-University  of  Connecticut, 
http://www.gifted.uconn.edu/siegle/research/Correlation/alphaleve.htm  (accessed  7  April 
2012). 

“  Acastat  Software,  Research  Tools  and  Instructional  Aids, 
http://www.acastat.com/Statbook/correlation.htm  (accessed  7  April  2012). 
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Table  1.  Characterizations  of  Pearson  R 

Very  high  correlation 

.9  to  1 

High  correlation 

.7  to  .9 

Moderate  correlation 

.5  to  .7 

Low  correlation 

.3  to  .5 

Little  if  any  correlation 

0  to  .3 

Source :  Created  by  author,  data  from  Acastat  Software.  “Research  Tools  and 
Instructional  Aids,”  http://www.acastat.com/Statbook/correlation.htm  (accessed  7  April 
2012). 


The  decision  criteria  for  satisfying  the  null  hypothesis  that  maintenance  metrics 
correlate  to  mishaps  experienced  is  comparing  the  final  correlation  coefficients  to  the 
Pearson  critical  values  table  for  the  sample  size  of  9  years  (or  N)  at  the  .05  confidence 
interval,  or  stated  alternately  the  results  are  expected  to  be  accurate  95  percent  of  the 
time.29 


29 

"  Del,Siegle,  Critical  Values  of  the  Pearson  Product-Moment  Correlation 
Coefficient,  http://www.gifted.uconn.edu/siegle/research/Correlation/corrchrt.htm 
(accessed  7  April  2012). 
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Table  2.  Pearson  Critical  Values  for  Two-Tailed  Probabilities 

N 

0.1 

4 

0.900 

0.950 

0.990 

5 

0.805 

0.878 

0.959 

6 

0.729 

0.811 

0.917 

7 

0.669 

0.754 

0.875 

8 

0.621 

0.707 

0.834 

9 

0.582 

0.666 

0.798 

10 

0.549 

0.632 

0.765 

Source :  Adapted  by  author,  data  from  Del  Siegle,  Critical  Values  of  the  Pearson 
Product-Moment  Correlation  Coefficient,  http://www.gifted.uconn.edu/ 
siegle/research/Correlation/corrchrt.htm  (accessed  7  April  2012). 


Metrics  data  variation 

No  effort  was  made  to  interpret  the  meaning  of  variances  in  metrics  data  over  a 
period  of  years  or  the  number  of  mishaps  (or  lack  thereof)  over  the  same  period  for  the 
units  measured.  Leadership,  maintenance  discipline,  weather,  organization  operations 
tempo,  unit  mission,  deployments,  exercises,  modification,  fleet  utilization  and  many 
other  factors  can  affect  a  unit’s  perfonnance  with  respect  to  its  metrics.  For  the  purpose 
of  this  study  it  is  assumed  that  all  units  are  affected  by  these  factors  equally. 
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CHAPTER  4 


ANALYSIS 

Understanding  the  Metrics 

While  the  USAF  tracks  over  thirty  different  aircraft  maintenance  related  metrics, 
this  paper  will  focus  on  the  nine  most  commonly  emphasized  metrics  in  the  F-16 
community.  The  primary  maintenance  metrics  tracked  are:  Mission  Capable  rate  (MC), 
Total  Non-Mission  Capable  rate  Maintenance  (TNMCM),  Total  Non-Mission  Capable 
rate  Supply  (TNMCS),  Break  rate,  Abort  rate,  8-hour  Fix  rate,  Cannibilization  rate, 
Repeat/Recur  rate,  and  Flying  Scheduling  Effectiveness  (FSE)  rate.  The  Mission  Capable 
rate  or  MC  is  the  percentage  of  the  unit’s  possessed  hours  for  aircraft  that  can  fly  at  least 
one  assigned  mission.  Aircraft  can  often  be  on  an  installation  and  be  possessed  by 
another  unit  or  entity  while  awaiting  disposition  instructions  for  repair  from  an 
engineering  authority,  official  transfer  of  possession  from  one  unit  to  another,  or  while 
being  repaired  or  modified  by  another  organization.  While  the  aircraft  experiences  one  of 
these  (or  other  similar  situations)  events,  this  time  does  not  factor  into  the  unit’s  Mission 
Capable  rate.  The  MC  rate  is  the  most  basic  of  metrics  as  it  lets  leadership  at  all  levels 
know  the  number  of  aircraft  available  to  fly  to  meet  everyday  training  sorties  or  potential 
contingency  operations.  MC  rate  is  usually  the  first  maintenance  metric  talked  about  in 

•3  1 

leadership  meetings  across  the  wing. 

30 

Department  of  the  Air  Force,  Air  Force  Instruction  21-103,  Equipment 
Inventory,  Status  And  Utilization  Reporting,  para  A2.2  (Washington,  DC:  Government 
Printing  Office,  2011),  www.e-publishing.af.mil  (accessed  15  May  2012). 

3  1 

Author’s  personal  experience  in  aircraft  maintenance  from  1989  to  201 1. 
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The  Total  Non-Mission  Capable  Maintenance  rate  or  TNMCM  is  the  percentage 

32 

of  possessed  hours  for  aircraft  that  cannot  fly  any  assigned  mission  due  to  maintenance. 
These  are  the  aircraft  actively  being  repaired,  regardless  of  the  length  of  time  the  repair 
takes  to  accomplish.  This  metric  is  almost  exclusively  controlled  by  the  maintenance 
organization  and  is  the  normal  cost  of  doing  business  in  a  maintenance  organization  as 
aircraft  will  inevitably  break  and  must  be  repaired. 

The  Total  Non-Mission  Capable  Supply  rate  or  TNMCS  is  the  percentage  of 
possessed  hours  for  aircraft  that  cannot  fly  any  assigned  mission  due  to  lack  of  parts. 

This  metric  is  the  organizational  level  grade  for  the  entire  supply  chain.  TNMCM  time 
stops  when  parts  are  ordered  against  an  aircraft  and  TNMCS  time  starts.  This  metric, 
combined  with  the  MC  and  TNMCM  numbers  will  account  for  the  entire  possessed  time 
for  the  local  aircraft  fleet. 

The  Abort  rate  is  the  number  of  sorties  aborted  for  whatever  reason,  which 
includes  air  aborts  and  ground  aborts.34  The  Abort  rate  is  computed  by  adding  all  air  and 
ground  or  training  aborts  together  and  dividing  by  the  total  number  of  sorties  attempted 
(flown  and  ground  aborts),  multiplied  by  100.  This  rate  is  usually  used  when  planning  the 
number  of  aircraft  to  schedule  for  a  successful  number  of  sorties  for  a  given  period.  For 
example,  if  the  historical  Abort  rate  is  5  percent,  for  every  20  aircraft  scheduled  to  fly, 

32 

Department  of  the  Air  Force,  Air  Force  Instruction  21-103,  Equipment 
Inventory,  Status  And  Utilization  Reporting  para  A2.5.2  (Washington,  DC:  Government 
Printing  Office,  2011),  www.e-publishing.af.mil  (accessed  15  May  2012). 

33Ibid.,  para  A2.5.1. 

34  Air  Force  Logistics  Management  Agency,  Metrics  Handbook  for  Maintenance 
Leaders,  http://www.aflma.hq.af.mil/shared/media/document/AFD-100107-055.pdf 
(accessed  15  May  2012),  26. 
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the  unit  expects  at  least  one  of  those  aircraft  will  not  fly  the  scheduled  sortie.  Based  on 
information,  extra  aircraft  are  provided  for  each  scheduled  period  to  mitigate  this 
potential  loss. 

The  Break  rate  is  the  number  of  aircraft  landing  with  a  grounding  discrepancy  (a 
discrepancy  preventing  the  aircraft  from  flying)  per  total  number  of  sorties  and  derived 
by  dividing  the  number  of  sorties  landing  broken  by  the  number  of  sorties  flown  for  a 
given  time  period,  multiplied  by  100.  ~  The  Break  rate  allows  leaders  to  anticipate  the 
normal  maintenance  attrition  for  an  organization.  In  line  with  the  historic  Break  rate,  the 
wing  or  squadron  will  usually  schedule  fewer  sorties  in  the  second  and  subsequent 
periods  of  the  flying  day,  knowing  that  a  certain  percentage  will  break  after  a  sortie. 

The  8-hour  Fix  rate  is  the  number  of  aircraft  landing  from  flight  with  grounding 
discrepancies  repaired  versus  the  total  number  of  aircraft  with  grounding  discrepancies 
and  is  derived  by  dividing  the  number  of  discrepancy  aircraft  repaired  by  the  number  of 
discrepancy  aircraft  landing  broken  for  a  given  time  period  multiplied  by  100.  This  is 
the  metric  used  to  measure  a  unit’s  ability  to  quickly  repair  aircraft  for  the  next  flying 
period.  It  is  usually  controlled  by  senior  non-commissioned  officers  in  charge  of  the 
unit’s  aircraft  technicians  at  the  organizational  level  of  maintenance. 

The  Cannibalization  rate  is  the  number  of  cannibalizations  (removal  of  an  aircraft 
part  from  another  aircraft  instead  of  receiving  the  part  from  the  supply  system)  that  occur 
per  sortie  and  is  derived  by  adding  all  aircraft  to  aircraft  cannibalization  actions  and  all 
engine  to  aircraft  cannibalization  actions,  then  dividing  by  the  number  of  sorties  flown 

35Ibid. 

36Ibid. 
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for  a  given  time  period,  multiplied  by  100.  This  metric  shows  the  amount  of  work  done 
by  organizational  level  maintenance  technicians  to  make  up  for  shortages  in  the  supply 
system.  Cannibalization  actions  are  particularly  inefficient,  as  a  serviceable  part  is 
removed  from  one  aircraft,  transported,  and  re-installed  in  another  aircraft,  instead  of  a 
part  being  issued  from  the  supply  system.  This  essentially  doubles  the  amount  of  work 
required  to  fix  a  single  discrepancy  as  the  part  is  removed  twice  (original  and  donor 
aircraft)  and  re-installed  twice  (original  immediately,  donor  when  the  part  comes  in  from 
supply).  It  also  can  significantly  extend  the  repair  time  on  an  aircraft. 

The  Repeat  and  Recur  rate  is  the  total  number  of  repeat  discrepancies  added  to  the 
total  number  of  recurring  discrepancies,  then  divided  by  the  total  number  of  pilot  reported 
discrepancies  for  a  given  period  of  time,  and  multiplied  by  100.  This  rate  effectively 
measures  how  well  a  maintenance  organization  fixes  discrepancies  the  first  time.  High 
repeat/recur  rates  are  historically  attributed  to  a  lack  of  thorough  trouble-shooting  of 
discrepancies,  pressure  to  quickly  repair  aircraft  for  subsequent  sorties  on  the  flying 
schedule,  or  a  lack  of  experienced,  qualified,  or  trained  aircraft  maintenance 
technicians.39 

The  Flying  Scheduling  Effectiveness  rate  or  FSE  is  adjusted  sorties  scheduled 
minus  the  number  of  chargeable  deviations,  divided  by  the  number  of  adjusted  sorties 


T7 

Air  Force  Logistics  Management  Agency,  Metrics  Handbook  for  Maintenance 
Leaders,  http://www.aflma.hq.af.mil/shared/media/document/AFD-100107-055.pdf 
(accessed  15  May  2012),  26. 

38Ibid. 

39Ibid. 
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scheduled,  for  a  given  time  period,  multiplied  by  100. 40  This  particular  rate  is  very 
important  as  it  shows  the  level  of  deviation  from  the  planned  operations  and  maintenance 
schedule.  As  the  rate  declines,  the  level  of  effort  to  provide  aircraft  to  the  flying  schedule 
increases.  Switching  aircraft  for  a  particular  sortie  may  seem  a  benign  event,  but  this 
requires  that  another  aircraft  be  prepared  for  the  flying  schedule  and  may  require 
additional  personnel  and  time  to  achieve  the  sortie.  The  cascading  effects  of  multiple 
aircraft  swaps  for  broken  aircraft  or  pilot  training  objectives  can  quickly  overwhelm  an 
organization. 

Applying  the  Metrics  to  Daily  Operations 
A  USAF  F-16  will  typically  begin  the  day  in  Fully  Mission  Capable  status.  As  the 
aircraft  maintenance  staff  arrives  for  the  day,  the  plane’s  status,  as  tracked  for  availability 
purposes,  can  change  throughout  the  day.41  If  the  aircraft  is  scheduled  to  fly  on  a  given 
day,  the  normal  process  begins  with  a  servicing  crew  checking  important  fluid  levels, 
such  as  engine  oil,  aviation  fuel  levels,  and  hydraulic  fluid  levels  of  the  aircraft. 
Additionally,  gaseous  pressure  levels,  including  the  pneumatic  pressure  of  the  aircraft’s 
tires,  and  various  bottles  of  nitrogen  are  checked  as  well.  If  any  of  these  fail  to  meet 
standards,  the  proper  aircraft  forms  are  annotated  appropriately  and  the  status  of  the 


40  Air  Force  Logistics  Management  Agency,  Metrics  Handbook  for  Maintenance 
Leaders,  http://www.aflma.hq.af.mil/shared/media/document/AFD-100107-055.pdf 
(accessed  15  May  2012),  27. 

41This  scenario  throughout  this  section  is  based  on  the  author’s  personal 
experience  working  with  F- 16s  in  three  different  maintenance  organizations  and  12  years 
of  experience. 


23 


aircraft  changes  to  Not  Mission  Capable  Maintenance  or  NMCM.  The  aircraft  remains  in 
that  status  until  the  maintenance  discrepancy  is  repaired  or  serviced. 

Once  the  aircraft  is  ready  for  flight  and  the  aircraft  forms  are  in  order,  the 
maintenance  crew  will  inform  the  operations  squadron  flight  desk,  the  maintenance 
operations  center,  and  finally,  the  pilot.  Once  the  pilot  arrives  at  the  aircraft  the  sortie  has 
officially  started  and  the  pilot  assumes  control  of  the  aircraft  for  its  sortie  or  flight.  The 
pilot  will  then  accomplish  his  or  her  pre-flight  inspection.  If  any  grounding  discrepancies 
are  discovered,  the  aircraft  sortie  is  considered  aborted,  and  the  pilot  will  most  likely 
proceed  to  another  aircraft.  The  current  aircraft’s  forms  will  be  annotated  and  its  status 
reported  as  NMCM.  Most  often,  however,  the  pilot  climbs  into  the  cockpit,  starts  the 
engine,  proceeds  with  additional  powered  pre-flight  operational  checks,  and  taxis  out  to 
the  end  of  runway  for  last  chance  checks  prior  to  takeoff.  If  at  any  time  in  the  process  a 
discrepancy  is  found  with  the  aircraft,  the  pilot  will  normally  return  to  the  aircraft  parking 
spot  to  allow  maintenance  personnel  the  opportunity  to  fix  the  discrepancy  and  allow  the 
pilot  to  make  the  sortie.  If  the  discrepancy  cannot  be  fixed  in  sufficient  time  to  allow  the 
pilot  to  make  the  sortie  in  the  allotted  time  frame,  the  pilot  will  proceed  to  another 
aircraft  or  back  to  the  squadron’s  building  and  the  aircraft’s  forms  will  be  annotated  and 
its  status  reported  as  NMCM.  For  tracking  purposes,  as  described  above,  an  aircraft 


42 

“Normally  each  flight  counts  as  one  sortie,  however,  occasionally  more  than  one 
sortie  can  be  accomplished  on  a  given  flight.  One  such  example  is  the  “Fight-Tank-Fight” 
scenario  where  a  pilot  may  engage  an  adversary  in  a  training  scenario  during  the  first 
sortie,  rendezvous  with  an  airbom  fuel  tanker  to  receive  additional  fuel,  then  engage  in 
another  sortie  for  a  different  training  objective.  In  this  instance,  two  sorties  may  be 
recorded  for  one  flight. 
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failing  to  meet  a  scheduled  sortie  after  the  pilot  has  arrived  at  the  aircraft  is  considered  a 
ground  abort.43 

If  the  aircraft  passes  all  checks,  the  pilot  will  proceed  to  takeoff  and  fly  the 
planned  sortie.  Once  airborne,  if  the  aircraft  develops  an  unsafe  condition,  the  sortie  is 
aborted  and  the  pilot  returns  to  base,  which  is  known  as  an  air  abort  for  tracking 
purposes.  Any  aircraft  discrepancy  causing  the  pilot  to  fail  to  successfully  fly  a  sortie 
once  he  or  she  arrives  at  the  aircraft,  is  considered  an  abort,  regardless  of  whether  or  not 
it  flew. 


Most  sorties  are  successful,  but  may  still  have  aircraft  discrepancies  of  a  non¬ 
emergency  nature.  These  pilot  reported  discrepancies  are  annotated  by  the  pilot  in  the 
aircraft  forms  and  designated  as  one  of  three  types  by  maintenance  personnel: 

A  grounding  condition  is  annotated  by  a  red  X  marking  in  the  aircraft 
forms  and  means  the  aircraft  is  not  fit  to  fly  in  its  present  condition.  This  aircraft’s 
forms  are  annotated  appropriately  with  the  specific  discrepancy  and  its  status 
reported  as  NMCM  until  maintenance  personnel  repair  the  discrepancy. A  non¬ 
grounding  condition  is  annotated  by  a  red  diagonal  mark  in  the  aircraft  forms  and 
represents  an  unsatisfactory  condition,  but  not  an  unsafe  condition.  This  may  or 
may  not  be  repaired  prior  to  the  next  scheduled  flight  but  does  not  render  the 
aircraft  NMCM.  An  informational  notation  may  also  be  made  by  the  pilot  to  note 
a  condition  not  normally  encountered  or  one  unfamiliar  to  the  pilot.  Maintenance 
technicians  may  or  may  not  repair  or  research  the  note  before  the  next  scheduled 
flight  and  this  note  does  not  render  the  aircraft  NMCM.44 


43  Air  Force  Logistics  Management  Agency,  Metrics  Handbook  for  Maintenance 
Leaders,  http://www.aflma.hq.af.mil/shared/media/document/AFD-100107-055.pdf 
(accessed  15  May  2012),  26. 

44Department  of  the  Air  Force,  Technical  Order  00-20-1 ,  Aerospace  Equipment 
Maintenance  Inspection,  Documentation,  Policies,  And  Procedures,  http://www. 
tinker.af.mil/shared/media/document/AFD-061220-046.pdf  (accessed  15  May  2012), 
para  4.1.2. 
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Aircraft  discrepancies  are  also  tracked  for  repeat  or  recurrence.  A  particular 
discrepancy,  for  example  one  involving  radios,  may  have  occurred  on  the  prior  flight  of 
the  aircraft.  This  is  annotated  in  the  aircraft  forms  as  a  repeat  discrepancy.  If  the  same 
discrepancy  occurred  in  the  last  five  aircraft  sorties,  the  discrepancy  is  annotated  as  a 
“recur”  or  recurring  discrepancy.  This  type  of  discrepancy  is  particularly  important  to  the 
maintenance  community.  It  may  indicate  numerous  problems  with  an  aircraft  system, 
aircraft  parts,  maintenance  repair  issues,  or  even  maintenance  training  shortcomings.  The 
repeat  and  recur  rates  are  one  of  the  more  important  indicators  for  maintenance  leaders. 

Another  important  indicator  is  the  8-hour  fix  rate.  Once  the  pilot  annotates  a 
grounding  condition  after  a  flight,  the  clock  starts  on  an  8-hour  window  to  measure  if  the 
maintenance  organization  can  fix  the  aircraft  within  that  time.45  The  purpose  behind  this 
metric  is  not  only  to  maintain  aircraft  availability  in  a  timely  manner,  but  during  a 
wartime  mission  when  sorties  are  normally  flown  around  the  clock,  rapid  repair  of 
aircraft  becomes  increasingly  important.  The  downside  to  rapid  repair  of  aircraft  may 
come  in  failure  to  fully  repair  or  temporarily  fix  an  aircraft  discrepancy.  This  can  result  in 
a  repeat  or  recurring  discrepancy,  or  worse,  an  aircraft  mishap. 

Upon  successful  completion  of  repairs  to  the  aircraft,  it  is  again  returned  to  a  fully 
mission  capable  status,  annotated  as  such  in  the  maintenance  collection  system  and 
prepared  for  the  next  day's  flying. 


45Air  Force  Logistics  Management  Agency,  Metrics  Handbook  for  Maintenance 
Leaders,  http://www.aflma.hq.af.mil/shared/media/document/AFD-100107-055.pdf 
(accessed  15  May  2012),  26. 
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USAF  Mishap  Criteria 

The  reporting  criteria  that  help  the  reader  understand  the  information  presented 


follows,  to  include  the  criteria  for  a  Class  A  mishap  and  the  minimum  required 
investigation  body. 

1.10.1.  Class  A  Mishap.  A  mishap  resulting  in  one  or  more  of  the  following: 

1.10.1.1.  Direct  mishap  cost  totaling  $1,000,000  or  more. 

1 . 10. 1 .2.  A  fatality  or  permanent  total  disability. 

1.10.1.3.  Destruction  of  a  DoD  aircraft  (Attachment  1).  NOTE:  A  destroyed 
UAV/UAS  is  not  a  Class  A  mishap  unless  the  criteria  in  paragraphs  1.10.1.1.  or 
1.10.1.2.  are  met.46 


Table  3.  Investigation  Membership  Minimum  Requirements. 


1 

Class  A  (Destroyed  Aircraft, 
Fatality,  or  Permanent  Total 
Disability)  (Note  1) 

Board  President,  Investigating 
Officer,  AFSC  Rep  (Note  2), 
Maintenance  Member,  Medical 
Member  Pilot  Member, 

Recorder 

2 

Class  A  (Other) 

Board  President  Investigating 
Officer  1  other  Primary  Member, 
Recorder 

3 

Class  A  or  Class  B  (Engine- 
Confined)  (Note  3) 

Single  Investigating  Officer  until 
failure  mode  is  determined,  then 
comply  with  Note  4 

4 

Class  B  (Other) 

Board  President  Investigating 
Officer 

or  Investigating  Officer 

1  other  Primary  Member 

Source :  Adapted  by  the  author  from  USAF  manual  91-223,  Aviation  Safety  Investigations 
and  Reports  (Washington,  DC:  Government  Printing  Office,  2006),  www.e.publishing. 
af.mil  (accessed  15  May  2012). 


46Department  of  the  Air  Force,  Air  Force  Instruction  91-204,  Safety  Investigations 
and  Reports  (Washington,  DC:  Government  Printing  Office,  2011),  www.e- 
publishing.af.mil  (accessed  17  May  2012). 
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Analysis  of  the  F-16  Fleet 

Analysis  begins  with  the  MC  rate  for  the  entire  F-16  fleet  for  the  FYs  2001 
through  2010  and  the  66  Class  A  mishaps  across  those  years,  as  shown  in  figure  1. 


Figure  1.  F-16  MC  vc  Mishaps  FY01-FY10 


Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


Figure  1  shows  a  low  of  80. 1  percent  in  2001  and  a  high  of  84.5  percent  in  2005, 
with  the  MC  rate  apparently  having  no  discernible  correlation  with  the  number  of 
mishaps  experienced  across  the  USAF,  which  ranged  from  a  high  of  13  mishaps  in  FY 
(FY)  2001  to  a  low  of  2  in  FY  2004.  To  further  illustrate  this,  the  MC  rate  was  arranged 
in  ascending  order  and  the  corresponding  number  of  mishaps  was  plotted  on  a 
scattergram  as  shown  in  figure  2.  Scattergrams  are  used  throughout  this  chapter  to 
provide  a  visual  representation  of  the  trend  line  generated  by  a  mathematical  formula 
known  as  Pearson’s  Coefficient,  discussed  in  chapter  3.  The  most  important  part  of  the 
scattergram  is  the  actual  coefficient  number  in  the  upper  right  hand  corner  of  each  graph. 
The  closer  that  number  is  to  1 ,  the  higher  the  correlation.  Conversely,  the  closer  the 
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number  is  to  zero,  the  lower  the  correlation.  The  distance  of  the  data  points  away  from 
the  trend  line  also  carries  meaning,  showing  how  much  variance  in  the  data  was 
experienced  and  how  reliable  the  data  is  with  respect  to  the  correlation  being  measured. 
The  resultant  trend  line  and  coefficient  of  correlation  for  the  MC  rate  was  .02773, 
statistically  insignificant. 


Figure  2.  MC  rate  scattergram 


Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


Next,  the  Total  Non-Mission  Capable  Maintenance  Rate,  or  the  amount  of  time 
aircraft  were  awaiting  or  being  repaired  and  not  capable  of  flying  assigned  missions  is 
depicted  in  figure  3  showing  the  rate  by  year  against  the  number  of  mishaps  experienced. 
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Auf  16  AU  F _16  TNMCM  vs  Mishaps  FY01-FY10 
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Figure  3.  All  F- 16  TNMCM  vs.  Mishaps  FY01-FY10 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


TNMCM 

•Mishaps 


From  a  low  of  10.9  percent  in  2005  to  a  high  of  13.6  percent  in  2008  and  2010, 
the  TNMCM  rate  also  fails  to  exhibit  any  correlation  with  the  number  of  mishaps 
experienced.  The  scattergram  plot  results  as  noted  in  figure  4  show  a  very  insignificant 
correlation  coefficient  of  .02475. 
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Figure  4.  TNMCM  rate  scattergram 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


Next,  the  Total  Non-Mission  Capable  Supply  rates,  or  the  amount  of  time  aircraft 
were  awaiting  parts  to  be  repaired  as  shown  in  figure  5  for  the  entire  F-16  fleet  for  the 
FYs  2001  through  2010.  The  associated  scattergram  for  the  TNMCS  rate  shown  in  figure 
6  also  fails  to  show  any  significant  correlation  as  exhibited  by  the  coefficient  of  .17873. 
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A" F 16  All  F-16  TNMCS  vs.  Mishaps  FY01-FY10 
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Figure  5.  All  F-16  TNMCS  vs.  Mishaps  FY01-FY10 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


Figure  6.  TNMCS  scattergram 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 
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The  Break  rate  graph,  showing  the  percentage  of  the  time  that  an  aircraft  would 
land  with  a  grounding  discrepancy,  for  all  F-16  units  from  FY  2001  through  2010  is 
shown  in  figure  7.  The  scattergram  for  the  Break  rate  shown  in  figure  8,  depicts  an 
extremely  low  correlation  coefficient  of  .01958  and  shows  a  particulary  large  variance  in 
data  from  the  trend  lines  center. 


Figure  7.  All  F-16  Break  rate  vs.  Mishaps  FY01-FY10 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 
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Figure  8.  Break  rate  scattergram 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


The  8-hour  Fix  rate  graph,  showing  the  percentage  of  the  time  aircraft  with 
grounding  conditions  after  flight  were  repaired  within  the  eight  hour  standard,  shows  no 
apparent  correlation  with  the  number  of  mishaps  experienced  as  shown  in  figure  9.  The 
scattergram  of  the  same  8-hour  Fix  rate  data  shows  a  very  insignificant  correlation 
coefficient  of  .04918  in  figure  10. 
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A" F 16  All  F-16  FIX  vs.  Mishaps  FY01-FY10 
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Figure  9.  All  F-16  Fix  rate  vs.  Mishaps  FY01-FY10 

Source:  Created  by  author,  using  data  AFSAS,  https://afsas.kirtland.af.mil,  and  LIMS- 
EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


Figure  10.  8 -hour  Fix  rate  scattergram 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


The  Abort  rate,  showing  the  percentage  of  sorties  aborted  for  the  entire  F-16 
community  for  FYs  2001  through  2010  is  shown  in  figure  11. 
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Figure  11.  All  F-16  Abort  rate  vs.  Mishaps  FY01-FY10 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


2001  2002  2003  2004  2005  2006  2007  2008  2009  2010 


Figure  12.  Abort  rate  scattergram 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 
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The  correlation  coefficient  for  the  Abort  rate  is  an  insignificant  .05619  as  shown 
in  figure  12.  The  far  left  aberration  was  due  to  an  absence  of  data  available  for  this  metric 
in  FY  2009. 
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Figure  13.  All  F-16  Repeat  and  Recur  rate  vs.  Mishaps  FY01-FY10 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


The  Repeat  and  Recur  rate,  or  the  number  of  times  aircraft  landed  with  the  same 
discrepancy  from  the  last  flight  (repeat)  or  the  last  five  flights  (recur)  is  shown  in  figure 
13  for  the  entire  F-16  fleet  for  FYs  2001  through  2010.  This  Repeat  and  Recur 
scattergram  in  figure  14  shows  a  correlation  rate  of  .25842,  the  second  highest  in  this 
study,  although  still  not  a  significant  correlation  coefficient. 
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Figure  14.  Repeat  and  Recur  rate  scattergram 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


The  Cannibalization  rate,  or  the  number  of  times  aircraft  parts  were  removed  from 
another  aircraft  instead  of  being  issued  from  the  supply  system  to  repair  a  non-mission 
capable  aircraft  is  shown  in  figure  15. 
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Figure  15.  All  F-16  Cannibalization  rate  vs.  Mishaps  FYOl-FYlO 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 
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The  Cannibalization  scattergram  has  the  highest  coefficient  correlation  at  .26023 
for  the  entire  F-16  fleet  metrics.  While  this  number  represents  the  highest  correlation  in 
this  study,  it  is  still  far  from  being  considered  significant. 


Figure  16.  Cannibalization  rate  scattergram 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


The  Flying  Scheduling  Effectiveness  rate,  or  the  degree  to  which  the  aircraft 
flying  schedule  was  adhered  to,  is  displayed  in  figure  17  for  the  10  year  period.  The 
scattergram  for  the  FSE,  at  a  .1 1814  coefficient  correlation,  is  not  a  significant  number  as 
shown  in  figure  18. 
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Figure  17.  All  F-16  FSE  vs.  Mishaps  FY01-FY10 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


Figure  18.  FSE  rate  scattergram 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


Analysis  of  the  entire  F-16  fleet  from  FY  2001  through  2010  shows  no  metric 
with  a  correlation  coefficient  higher  than  .26023,  the  correlation  coefficient  associated 
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with  the  Cannibalization  rate  for  the  decade.  Using  table  1  from  chapter  3,  it  is  apparent 
this  is  in  the  “Little  if  any  correlation”  range. 

Comparison  of  individual  active-duty  units 
Nine  active  duty  units  were  compared  for  the  purposes  of  answering  the  research 
questions.  All  units  were  actively  flying  and  maintaining  F-16  for  the  period  of  FY  2000 
through  FY  2008.  Comparison  of  the  active  duty  units  begins  with  this  table  of  the  units, 
with  the  mishaps  plotted  for  the  corresponding  FYs: 


Table  4.  Active  duty  unit  mishaps  2000-2008 

FY 

2000 

2001 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

Totals 

Organization 

8  FW 

1 

1 

1 

3 

20  FW 

1 

1 

1 

1 

4 

31  FW 

1 

1 

2 

35  FW 

2 

1 

3 

51  FW 

1 

1 

2 

52  FW 

1 

1 

2 

56  FW 

2 

1 

1 

2 

1 

2 

2 

11 

57  FW 

1 

1 

1 

3 

388  FW 

1 

1 

4 

1 

7 

Totals 

4 

6 

4 

9 

3 

5 

4 

2 

37 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


The  total  number  of  Class  A  mishaps  for  the  period  studied  was  37.  There  were 

69  across  the  entire  USAF  F-16  fleet  during  this  period,  with  1 1  of  those  coming  from  the 

56th  FW,  the  Air  Education  and  Training  Command  base  responsible  for  initial  F-16  pilot 

training.  Again,  emphasizing  the  scope  and  limitations  of  the  paper,  no  attempt  to 
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ascertain  maintenance  cultural  effects,  unit  mission,  or  any  other  factors  contributing  to 
the  metrics  measured,  will  be  made. 

While  each  unit’s  metrics  have  been  plotted  and  can  be  viewed  in  the  attached 
tables,  the  only  scattergrams  and  histograms  displayed  in  this  chapter  will  emphasize  the 
high  and  low  correlations,  as  measured  by  Pearson’s  correlation  coefficient.  Apparent 
visual  correlations  in  the  plots  of  the  mishaps  against  the  metrics  as  they  occurred  for 
each  FY  may  or  may  not  represent  a  mathematical  correlation. 

MC  rate  coefficients  for  the  nine  units  studied  presented  no  discemable  correlation 
between  the  MC  rate  and  mishaps  experienced,  as  evidenced  by  table  5. 

Table  5.  Mission  Capable  rate  correlation 
[  Range  j 


0.0000033 

Low 

0.00128 

0.00668 

0.01043 

0.04513 

0.0953 

0.09692 

0.10196 

0.2486 

High 

0.067367033 

Average 

Source:  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 
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Each  coefficient  number  was  derived  from  comparison  of  the  unit’s  Mission 
Capable  rate  perfonnance  over  the  nine  years  with  the  number  of  mishaps  experienced, 
from  the  lowest  MC  rate  year  through  the  highest.  Even  the  highest  value  achieved  by  the 
35  FW  at  .2486  is  not  halfway  to  achieving  the  .666  value  required  from  the  Pearson 
critical  values  table  for  the  chosen  confidence  level  of  .95,  the  most  commonly  used 
confidence  level  in  statistical  studies.47 

The  next  metric  to  be  compared  is  the  TNMCM  metric,  as  displayed  in  table  6. 


Table  6.  Total  Non-mission  capable  maintenance  rate  correlation 
j  Range  i 


0.00018 

Low 

0.00041 

0.00119 

0.00939 

0.02681 

0.04851 

0.05926 

0.17351 

0.20098 

High 

0.057804444 

Average 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


47Del  Siegle,  Confidence  Interval,  http://www.gifted.uconn.edu/siegle/research/ 
Samples/Confidencelnterval.htm  (accessed  7  April  2012) 
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The  average  coefficient  for  correlation  for  this  metric  was  .057804444.  This 
indicates  an  almost  complete  lack  of  correlation  between  the  TNMCM  metric  and 
number  of  mishaps  experienced  in  the  sample  group. 

The  TNMCS  metric  correlation  average  was  .178087778  as  shown  in  the  table  7. 
Again,  while  a  significantly  higher  number  than  the  previous  two  coefficients  and  the 
highest  of  the  active  duty  correlations,  statistically  this  is  a  very  low  correlation 
represented  by  this  average.  Of  note  is  the  sample  active-duty  units  TNMCS  rate  almost 
matches  perfectly  the  correlation  rate  for  the  entire  F-16  fleet  in  the  study. 


Table  7.  Total  Non-mission  capable  supply  rate  correlation 
I  Range  ) 


0.00018 

Low 

0.00187 

0.04556 

0.08988 

0.16788 

0.2025 

0.24863 

0.35953 

0.48676 

High 

0.178087778 

Average 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 
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The  next  metric  reviewed  was  the  Break  rate,  as  shown  in  the  table  8.  The  Break 


rate  represented  an  extremely  low  correlation  rate  from  the  sample  group. 


Table  8.  Break  rate  correlation 
I  Range  I 


0.00029 

Low 

0.00059 

0.00153 

0.01269 

0.01289 

0.03444 

0.04037 

0.04234 

0.06517 

High 

0.023367778 

Average 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


The  8-hour  fix  rate  correlation  coefficient  was.  103825556.  This  is  also  in  the  very 
low  correlation  range. 
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Table  9.  8  hour  fix  rate  correlation 

Range 

0.00193  Low 

0.00201 
0.00365 
0.01693 
0.0672 
0.08698 
0.17995 
0.19741 

0.37837  High 

0.103825556  Average 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 

The  Abort  rate  measured  had  a  very  low  correlation  coefficient  of  .053953  111. 
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Table  10.  Abort  rate  correlation 
Range 

0.000028  Low 

0.00915 
0.00919 
0.01254 
0.01258 
0.02777 
0.03536 
0.04788 

0.33108  High 

0.053953111  Average 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 

The  Flying  Scheduling  Effectiveness  rate  was  a  very  low  .079908067  as  depicted 
in  table  1 1 . 
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Table  11.  Flying  Scheduling  Effectiveness  correlation 

Range 

0.0000026  Low 

0.00432 
0.00601 
0.00865 
0.01221 
0.08033 
0.0845 
0.14594 

0.37721  High 

0.079908067  Average 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 

The  Cannibalization  rate’s  correlation  coefficient  was  .1652,  the  second  highest  of 
the  nine  active-duty  units  sampled  as  shown  in  table  12. 
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Table  12.  Cannibalization  rate  correlation 
Range 

0.00074  Low 

0.00235 
0.00345 
0.03483 
0.0382 
0.03887 
0.17025 
0.52409 

0.67402  High 

0.1652  Average 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 

The  Repeat/Recur  rate  was  .0730911 1 1  for  the  period  sampled,  in  table  13  and  a 
statistically  insignificant  correlation  coefficient. 
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Table  13.  Repeat  and  Recur  correlation 
I  Range  j 


0.00358 

Low 

0.01228 

0.01429 

0.03585 

0.05213 

0.0887 

0.1087 

0.15249 

0.1898 

High 

0.073091111 

Average 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 


The  comparison  of  the  nine  active  duty  units  produced  a  correlation  coefficient  no 
higher  than  the  .178087  associated  with  the  TNMCS  rate.  Most  of  the  coefficients  were 
significantly  lower  than  that  and  show  no  indication  of  correlation  to  unit  mishaps 
experienced. 
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CHAPTER  5 


CONCLUSIONS  AND  RECOMMENDATIONS 

Conclusion 

Answering  the  original  research  question,  “Does  a  discernible,  statistically 
relevant  relationship  exist  between  any  one  of  nine  specific  active-duty  F-16  maintenance 
perfonnance  indicators  and  active-duty  F-16  mishaps  experienced?”  can  only  lead  to  a 
negative  answer.  Based  upon  the  data  presented,  and  using  common  statistical  practices, 
the  only  possible  conclusion  is  there  is  not  a  particular  metric  reviewed  in  this  study  that 
correlates  to  the  Class  A  mishaps  experienced  by  the  unit.  A  quick  summary  table  of  the 
Pearson  correlation  coefficients: 


Table  14.  Summarization  of  correlation  data 

Metric 

All  F-16s  FY  2001-2010 

9  Active  duty  bases 
FY2000-FY2008 

MC  rate 

.02773 

.067367 

TNMCM  rate 

.02475 

.057804 

TNMCS  rate 

.17873 

.178087 

Break  rate 

.01958 

.023367 

Fix  rate 

.04918 

.103825 

Abort  rate 

.05619 

.053953 

Rep/Rec  rate 

.25842 

.073091 

Cannibalization  rate 

.26023 

.1652 

FSE  rate 

.11814 

.079908 

Source :  Created  by  author,  using  data  from  AFSAS,  https://afsas.kirtland.af.mil,  and 
LIMS-EV,  https://www.my.af.mi1/faswsv/faswsv/Main.html#  (accessed  15  May  2012). 
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There  are  some  metrics  to  note  however.  The  Cannibalization  rate  and  the  Repeat 
and  Recur  rate  for  the  entire  F-16  fleet,  while  not  meeting  the  criteria  for  consideration 
under  the  low  correlation  as  specified  in  chapter  three  (.3  to  .5  for  low)  had  significantly 
higher  coefficients  than  did  any  other  metrics. 

The  significance  of  the  Cannibalization  metric  is  in  the  amount  of  extra  work 
higher  rates  indicate  for  the  average  maintenance  technician  working  on  aircraft.  As  the 
TNMCS  rate  climbs,  meaning  more  aircraft  are  not  mission  capable  while  awaiting  parts 
from  supply,  the  Cannibalization  rate  usually  climbs  as  well  as  technicians  remove  parts 
from  donor  aircraft  in  order  to  meet  the  flying  schedule  requirements.  Additional  work 
across  the  same  number  of  personnel  may  result  in  rushed  technicians  or  less  than  stellar 
maintenance  practices  exhibited  while  trying  to  meet  the  flying  schedule.  It  is  also 
noteworthy  that  the  active-duty  sampled  metrics  had  the  Cannibalization  rate  a  very  close 
second  for  correlation  coefficients  at  .1652. 

The  significance  of  the  Repeat  and  Recur  rate  traditionally  points  to  training  or 
insufficient  experience  issues  within  the  maintenance  workforce.  As  aircraft 
discrepancies  repeat  or  recur,  the  level  of  attention  and  work  involved  increases  in  an 
attempt  to  find  the  source  of  the  discrepancy  and  to  rectify  the  situation.  Repeat  and 
Recur  discrepancies  in  and  of  themselves  may  not  present  a  safety  issue;  however  the 
tendency  toward  a  higher  rate  usually  indicates  underlying  problems  in  the  organization. 
The  higher  correlation  coefficient  in  this  area  could  also  point  to  the  same  issues. 

Another  issue  of  significance  is  the  relative  lack  of  congruity  between  the 
sampled  active-duty  bases  and  the  entire  F-16  fleet  for  the  overlapping  FYs  of  2001- 
2008.  While  the  TNMCS  rate  was  nearly  identical,  and  the  Break  and  Abort  rates  were  a 
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.03  and  .04  difference,  four  of  the  nine  rates  studied  had  a  greater  than  50  percent 
difference  from  the  entire  F-16  fleet  in  both  positive  and  negative  correlation  levels, 
however  slight.  This  indicates  the  study’s  focus  may  have  been  too  narrow  in  scope. 
Eliminating  Air  National  Guard  and  Air  Force  Reserve  units,  while  significantly  reducing 
the  number  of  units  and  metrics  subjected  to  data  manipulation  and  calculation,  may  have 
also  removed  the  requisite  sample  size  of  mishaps  for  a  statistically  relevant  study.  The 
overall  measurements  for  the  entire  F-16  fleet  for  the  FYs  2001  through  2010  most  likely 
provided  a  more  accurate  representation  of  the  value  of  using  metrics  for  predicting 
mishaps. 

Overall  however,  the  nine  metrics  studied,  used  extensively  throughout  the  USAF 
and  worthwhile  for  indicating  various  circumstances  and  illuminating  potential  problems, 
do  not  appear  to  have  any  particular  value  in  predicting  the  chances  of  an  F-16  unit 
experiencing  a  Class  A  mishap.  This  is  the  case  regardless  of  consideration  of  the  entire 
fleet  or  a  smaller,  more  controlled  sample  of  active  duty  units. 

Recommendations 

While  this  study  failed  to  find  the  “silver  bullet”  of  mishap  prevention  through 
maintenance  metrics  study,  the  positive  correlation  between  the  Cannibalization  and 
Repeat  and  Recur  rates  may  point  to  a  slight  value  in  a  more  far  reaching  study 
specifically  focused  on  these  two  rates  and  perhaps  across  entire  fleets  using  aggregate 
numbers  from  both  active  duty  and  reserve  components.  One  potential  method  could 
utilize  the  summaries  available  from  the  USAF  Safety  Center  combined  with  the  annual 
metrics  reports  to  accomplish  a  “quick-look”  comparison  and  correlation  detennination  in 

an  effort  toward  increasing  USAF  aviation  safety. 
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